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One of the biggest problems of any program aimed at social change 
is feedback -- reliable, useful information about how well it is accom- 
plishing its objectives and where it should direct its energies. Only 
through feedback can a program adapt to conditions, or improve. Feed- 
back is especially important under certain conditions -» when a pro- 
gram is new, the people or environment unfamiliar, the number of per- 
sons with whom it must deal large ^ or the variety of circumstances in 
which it must function diverse. These are often the conditions for 
Peace Corps as well as other U.S .-supported efforts to help a developing 
country modernize some aspect of its life. And under such conditions, 
feedback is often extremely hard to get. This report describes some of 
the procedures we developed during two years of research to provide 
feedback for the Peace Corps Educational Television (ETV) Project in 
Colombf The ETV Project faced all of the conditions which make 
feedback important and difficult. Hopefully, our experience, gained 
by trial and error, will be of help in obtaining feedback in the future 
for the project in Colombia, or for other kinds of modernization 
programs abroad. 

The Special Nature of This Report 

Unlike the other nine reports in this series on our research on the 
ETV Project, this report is not devoted to the actual results of 
research. Instead, it is concerned with how some of those results 
were obtained. It is concerned with the methods and techniques we 
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found most successful. However, it is not a complete review of all the 
research procedures we used. It is only concerned with those which 
provided information of direct and insmediate utility for making decisions 
about what the project should do next. It also concentrates on those 
procedures which could be used readily by someone with a minimum of 
research experience, and which do not require extensive material or 
equipment. This is one of the important benefits of research in addition 
to data and findings -- it results in the devising of tools which others 
can apply later with much more ease and far less time and expense than 
were necessary for their development. Because they have been used before, 
and have proved successful, many of the procedures described in this 
report could be used again with little or no modification. 

Some Background 

The feedback problem, and oi-.r attempted solutions, can be better 
understood after a brief review of the operations of the ETV Project 
itself. The project faced two major tasks in becoming a functioning 
success. One was the production of televised instruction for pupils in 
public primary schools, and for their teachers. The other was the setting 
up of a network of schools with television to receive the instruction. 
Because the use of ETV in the school turned out to demand so much that 
was new for schools and teachers, ETV involved far more than the mere 
installation of TV sets — it also required the training of teachers 
in how to teach in conjunction with television, the reorganization of 
schools to meet the rigid schedules imposed by ETV, and often the 
mobilization of the school and sometimes of the community to make the 
school suitable for television — for adequate wiring, electricity 
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supply, security of the set from thievery, and a suitable room for 
viewing , 

ivhen the ETv Project began in early 1964, it was telecasting 10 
different courses tc primary school pupils, with each consisting of two 
15 minute televised lessons each week for a weekly total of 300 minutes 
of television, and some 200 schools, 1,000 teachers, and 38,000 pupils 
were participating. By the end of 1965, when our research ended, 15 
different courses were being telecast for a weekly total of 450 minutes 
of television, and some 925 schools, 7,000 teachers, and 260,000 pupils 
were participating. Moreover, special programs and courses were tele- 
cast for the in-service training of primary school teachers. At the 
beginning, only schools in Bogota, Colombia’s capital, and the surrounding 
Department (state) of Cundinamarca were included. By the end of 1965, 
schools in six other Departments were included. During the two years, 
the number of Volunteers working in the ETV Project ranged from about 
60 to 85 (excluding the overlap of terminating and replacement Volun- 
teers), Except for about half a dozen assigned to the installation 
and maintenance of TV sets in schools, these Volunteers were either 
concerned v/ith a) the production of televised instruction, or b) the 
effective use of the televised instruction in the schools by the class- 
room teachers, v;ho v/ere expected to build their o\m teaching around 
the ’’core" of instruction provided by the telecasts. Throughout, roughly 
one-third of the Volunteers were in production, and slightly more than 
half to tX7o-thirds in ’’utilization’’ -- the term used to designate those 
working in the schools with teachers, because they were concerned with 
helping teachers to make the most effective use of televised instruction. 
Thus, on the one hand, the project was concerned with the success of a 
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broad and extensive televised curriculum, and on the other, \d.th the 
reception of that curriculum in schools, its effective treatment by 
teachers, and the success of the "utilization** Volunteers in promoting 
these goals. ^ 

The Organization of This Report 

We attempted to provide feedback about both of the project *s tasks. 

In Part j[ of this report, we outline some of the ways in which vie obtained 
feedback on the televised instruction. In Part II, we outline some of 
the ways in which we obtained fe' iback on the project *s impact in the 
schools and the Volunteers’ efforts to make ETV effective there. In 
each, a variety of feedback procedures are described. For each procedure, 
we have tried to explain not only what we did, but why we did it, and to 
cover some of the problems we encountered and the conditions necessary for 
its use. I'lhen a procedure involves a particular device, such as a set 
of questions in a survey questionnaire, we have included it verbatim, 
so that it might be more easily used again or adapted. We have also 
included examples of the ways in which the results might be set up for 
clear-cut presentation and interpretation. In Part III , we discuss 
some other feedback techniques which the ETV Project might employ; in 
Part IV . the flow and treatment of feedback necessaxry for its effective 
use; and, in Part V. the applicability of the techniques discussed for 
other kinds of projects. In Part VI . we list the places where a full 
report of the actual results obtained by our feedback techniques can 



er|c 



be found. 



-5- 




Part I: Feedback on the Televised Instruction 

One of the major concerns of any program is to obtain feedback so 
that it can improve its product. The primary product of the ETV Project 
was its televised instruction for pupils in public primary schools — 
courses in 10 subjects the first year, in 15 the second year, A secondary 
product was the televised instruction for the in-service training of 
teachers, which in the first year consisted of weekly telecasts on 
broad educational topics, such as child psychology, and in the second year 
involved “short courses” in some of the same subjects -- mathematics, 
social science, music, and the like — which the teachers themselves 
were to teach their pupils in conjunction xfith television. 

There are t \70 basic approaches to obtaining such feedback. One 
centers on finding out the reactions or a+'titudes tox/ard the product of 
those for x^hom it is intended. The other centers on determining the 
effectiveness, regardless of reactions or attitudes, of the product 
in achieving its goals. They are not likely to be independent in real 
life, for reactions might affect effectiveness, and effectiveness might 
affect reactions. However, the procedures for obtaining feedback of 
each sort are quite different. Each poses different problems, and has 
different advantages. We shall look at each in turn. However, x/e x^ill 
give the most extensive attention to feedback on teacher attitudes, and 
especially teacher attitudes tox-zard the televised instruction for pupils, 
because the procedures and even the instruments -- such as the question- 
naire — could be used again just as x/e used them xdLth little or no 
change. For other kinds of feedback, procedures, and especially instru- 
ments, xjould have to be fitted to the new situation. As a result, we 
can only discuss the procedures in general. 
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Use of Attitudes for Feedback on the Televised Instruction for Pupils 
The televised instruction for pupils actually involved tv7o groups — 
the pupils, who were expected to leam fTOm It, and the teachers, who 
were expected to build their ovm teaching around it. We did not try 
to obtain systematically any feedback on pupil attitudes toward the 
various courses. We knew they liked and enjoyed the telecasts. We 
observed numerous classes vzhile they watched the televised lessons, 
and V 76 alv?ays found that the children were entranced by the programs. 

At the slightest urging from the classroom teacher, they usually V70uld 
respond to the cues of the television teacher — to yell out a v7ord, 
repeat a phrase, or sing along. Since pupils in Colombian public 
schools come only from the lowest social strata, few if any had tele- 
vision at home. For almost all, school television was their first 
extended exposure to the medium, and for many, especially in rural 
areas, it was their initial exposure. As a result, none of the children 
had any developed vie\7ing habits, or notions about the kind of excite- 
ment or entertainment television might provide. Thus, there was no 
standard of diversion by which the instructional t Revision might 
suffer in comparison. Since attitudes or reactions arc of interest 
primarily because they presumably affect pupil motivation and interest — 
important factors for learning — it did not seem profitable to try to 
investigate differences in attitudes toward the various courses. The 
children were enthusiastic about all of them. If differences existed, 
they would have been so small that their effect x7ould have been negligible. 
During our two years of research on the ETV Project, not one utilization 
Volunteer, visiting classes daily, ever suggested that pupil attitudes 
toward a particular course interfered with its effectiveness, 
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Ths teachers were another matter. For each 15 minute televised 
lessen, they v;ere expected to provide 30 minutes of classroom teaching — 
15 minutes before, as "motivation,” and 15 minutes aften^ard, as "follow- 
up," Presumably, they v;ere to prepare this teaching in advance by using 
the Teacher Guides, which described the content of each telecast and 
gave suggestions for "motivation" and "follov7-up," They were far more 
likely than the pupils to have had some prior experience with television, 
and usually they had had a great deal of exposure to movies, vdiich are 
very popular as public entertainment in Colombia, Thus, they had seme 
kind of standards for judging the televised instruction. Far more 
important, they had, as teachers, definite standards for judging instruc- 
tion, They had ideas about what should be taught, and how it should be 
taught. Teaching was their job. From the teachers' vie^-Tpoint, they 
were experts. It seemed likely that they would have definite opinions 
about each of the courses. The utilization Volunteers frequently 
reported receiving complaints about one course or another. 

Much of the success of the project depended on the classroom teachers. 
If they liked, they could ignore the telecasts. Even if they did not 
ignore them altogether, they could diminish their effectiveness by dis- 
interested, clumsy, or haphazard "motivation" and "follow-up." In a 
very real sense, the fate of the project, once the televised lessons 
were produced, rested with the classroom teachers. It seemed very un- 
likely that a teacher would devote himself to inspired or even thorough 
teaching in conjunction xdLth a televised course which he held in low 
esteem. Moreover, with numerous courses being telecast, and each teacher 
teaching tv;o or three, it seemed unlikely that he would give the same 



attention to those he judged to be relatively inferior as he would do 
2 

to the others. As a result, we laade a considerable effort to obtain 
feedback on teacher attitudes tw/ard each of the various courses. 

We used a survey, and a self-completion objective questionnaire or 
rating form. We chose this method because, with different teachers in 
each of the five grades teaching different courses with television, it 
would have been impossible to reach enough in each grade by personal 
intervie\7ing for the results to be dependable. Moreover, we wanted 
each teacher to give her opinion about each course under identical con- 
ditions, so that the results for each course could be compared. Given 
the kind of information in which we were interested, we were afraid 
that personal interviewing might lead to too much variation in the way 
the teachers' opinions were obtained for the results for each course to 
be comparable. To overcome this variation, it would have been necessary 
to reach an even larger number vTith personal interviews than was 
necessary with a self -completion questionnaire. A self -completion 
questionnaire also had the advantage of easy and rapid coding and tabu- 
lation of the results. 

We conducted four surveys which contained questions about the 
specific courses during our two years of research, one at the end of 
each of the four semesters of instruction. Because of the quantity of 
televised instruction involved, the difficulties of sampling and of 
reaching the teachers with questionnaires, and the job of tabulating 
and analyzing results, it was feasible only to obtain sv/eeping judg- 

3 

ments on each course for a whole semester. Moreover, conversations 
with the utilization Volunteers, and interviews with teachers and other 
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school officials indicated that the teachers thought of a course as a 
whole. They were seldom ready VTith opinions on a single program. This 
was not surprising, since all the lessons for any one course were pro- 
duced by the same director with the same television teacher, and were 
similar in approach, style, and treatment of material. 

In our first survey, we used several questions on each course -- 
about whether the teachers thought it held the pupils* interest, whether 
it was visually attractive, and how '‘good” overall they thought it was. 

We found this was unnecessary. For determining the relative degree of 
approval which the teachers gave each course, a single, simple question 
proved sufficient. The replies to the others ranked the courses in 
about the same order, and gave no additional information. The single 
question was this: 

Taking into account all the aspects of the programs for 
this course* during this semester, how many of the tele- 
casts would you say have been excellent? 

All A few less than half 

Not all, but Only a few 

almost all 

A few more than Almost none 

half 

*the course was named in bold letters at 
the top of the page, and a separate page 
was used for each course. 

In analyzing the results, v;e used the per cent of teachers for each 
course who said all the telecasts for that course had been "excellent" 
as an index of teacher satisfaction. These per cents, it turned out, 
varied markedly among the courses, and it was possible to ranlc the courses 
from loV7 to high in regard to teacher satisfaction. Since the content, 
television teacher, director, and the courses themselves varied from 
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semester to semester, we did this for each of the four semesters. An 
example of the kind of data we obtained for one semester can be seen in 

the form of a bar graph in Figure 10:1. 

miat do these data tell us? They do not merely show that teacher 
satisfaction with the courses varied. They also show which of the 
courses were low, and which were high, in teacher esteem. As a result, 
the data provide, in regard to the courses, a ^ of priorities for 
action. They direct attention to those courses which the teachers 
found relatively inferior. Whether such courses are in fact inferior 
is another matter. The fact is that the teachers judge them so. As a 
consequence, it remains for project administrators to decide what should 
be done — vjhether the courses should be revised, or whether an informa- 
tion campaign to overcome the teachers' misgivings by explaining why 
the instruction is like it is should be launched. However, the data 
do make it clear that something should be done, and they do point to 
those courses which most demand attention. 

Outside of arbitrary and idealistic goals, such as 90 or 100 per 
cent, there is no objective criterion for what per cent of the teachers 
should say all the telecasts for a course were "excellent" before a 
course can be dismissed as posing no problem in regard to teacher atti- 
tudes. No matter how well-done a course, there will always bo a few 
who rd.ll find some of its telecasts imperfect, and there is no way to 
tell just how many this might be except possibly by educated guess 
and the education needed for the guess could be obtained only through 
looking at the results for surveys covering courses which had reached 
the upper limits of quality. For any given survey, of course, the top 
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per cent provides the goal which other courses could have been expected 
to equal. Thus, the only reasonable target is "high” and roughly equal 
per cents for all courses, with "high" defined as the rating received by 
the most favorably evaluated course. Examples of how top ratings can be 
used to set targets or goals empirically can be found where the full data 
on teacher course ratings are presented (see Part VI , this report). 

Important though such data are when there is a paucity of dependable 
Information about what the teachers thought of the courses, they do not 
tell anything about the source or cause of teacher dissatisfaction with 
a course. As a result, we also tried to get information from our surveys 
on what the teachers thought was inadequate about each course. 

In our first survey, as mentioned above, we Included questions similar 
in form to the one we have just discussed but dealing with specific aspects 
of the courses -- level of pupil interest aroused, visuals, and the like. 
These failed to provide any additional information, for the ranking of 
courses that resulted was much the same as for the question on how many 
of the telecasts had been "excellent," For our two surveys in 1965 con- 
cerned with the courses, we developed a check-list for teacher complaints. 
We called it a "dissatisfaction inventory" because it attempted to tally 
the specific dissatisfactions the teachers entertained about each course. 

It proved very successful. It provided specific information about each 
course, and did not merely duplicate the results on overall excellence. 

This was the check-list we used: 

Which of the following problems have you encountered with 
the programs for this course* during the semester? (Check 
as many as apply. If none apply, check none.) 

The programs cover too much material for the children 
to comprehend. 
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The children are not able to clearly see objects, 
maps, and things which are shoxTn, 

The programs teach little the classroom teacher 
cannot teach. 

The television teacher does not have a good persona- 
lity for television. 

The programs entertain, but teach very little. 

The children learn only from the pre-program 
"motivation” and the post -program "follow-up," 
and not from the program* 

The programs do not teach concepts, but only facts, 

*the course was named in bold 
letters at the top of the page, 
and a separate page was used 
for each course. 

The check-list immediately followed the question on overall excellence 
on the questionnaire. Thus, there was a check- list for each course. 

How did we construct this check-list? First, we gathered from the 
utilization Volunteers, teachers, and school officials about 200 com- 
plaints made by teachers about specific courses. For example, "Social 
Science for the fifth grade goes too fast. It covers too much, and the 
children can't keep up." Secoid, we catalogued these and grouped them 
by the kind of problem or dissatisfaction that was being expressed. 

Third, we wrote items covering each of the kinds of problems or dis- 
satisfactions. Thus, the check-list covered all the kinds of complaints 
the teachers were making, and in a standardized f^rm so that results 
for one '’ourse could be compared \d.th those for another. 

The check- list covered a broad range of issues. In the order of the 
items in the check-list, these included; a) the pace of the instruction; 
b) the adequacy of the visuals; c) whether the television was really 
giving instruction the classroom teacher x^as not easily able to give; 
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d) the acceptability of the television teacher; e) whether entertain- 
ment or "show biz" dominated the programs at the expense of instructional 
content; f) whether the television left too much to the classroom teacher; 
and, g) whether the television only presented facts, but not general 
principles. 

The latter item -- "The programs do not teach concepts, but only 
facts" -- requires a special note. It was included because several com- 
plaints v;ere catalogued phrased almost identically to the item. However, 
it is likely that the complainers meant that the telecasts did not pre- 
sent stock phrases which the children could memorize, a practice central 
to most Colombian teaching, and not at all what we would mean by the same 
x7ords. One of the goals of the ETV Project is to promote more imagina- 
tive teaching, and independent thinking on the part of pupils, and to 
discourage rote memorization. Nevertheless, the item was included because 
it seemed possible that there was some concern among the teachers about 
it. If there t^as widespread concern, then this would dictate giving 
some attention to whatever the teachers might have in mind. 

In examining the results from this check-list to find out what 
actually displeased the teachers about the various courses, \je subjected 
them to a variety of analyses. These are fully reported elsewhere. 

Among other things, however, they indicated that this item provided 
almost no usable information. The complaint was made with relatively 
moderate frequency (on the average for both semesters, by about 10 per 
cent of the teachers, making it fifth among the seven complaints), 

Hov’ever, unlike other complaints made v?ith similar average frequency, 
the frequency varied little among courses, so that particular courses 
could not be singled out. Moreover, when we determined the weight of 




-15- 



tb.e various complaints in the making of the overall ratings for tbe 
courses, we found that this complaint invariably played almost no role 
at all. From every perspective, then, it seemed not only ambiguous, 
but irrelevant. On the basis of these analyses, we would suggest that 
the item be dropped should the "dissatisfaction inventory" be used again. 

What kinds of information did this check-list provide? For one thing, 
merely constructing the check-list by collecting and grouping complaints 
resulted in a better picture of the kinds of factors which affected 
teacher attitudes toward the courses. More important, the results from 
the surveys indicated which of the dissatisfactions were common 

for the entire televised curriculum. To find this out, we took the per 
cent checking a particular complaint for each of the courses, added them, 
and divided the total by the number of courses. In doing so, we treated 
the per cent for any one complaint for each course as the best estimate 
for the "score" for that course, without comsidering differences in 
the number of teachers completing the questionnaire for each course. 

The result was a per cent for each complaint representing its average 
frequency for aU the courses. This led to a ranking of the complaints 
for the ETV curriculum, and some idea as to their frequency among the 
teachers. An example of the kind of data we obtained is shoi-Tn in 
Figure 10:2 . in the form of a bar graph . 

As can be seen, in this instance the most prominent concern among 

the teachers v/as the pace of the instruction -- they thought it was too 
fast. Next came the adequacy of the visuals. There was almost no con- 
cern over the telev? ion teacher or whether the television was giving 
instruction the classroom teacher could just as easily provide. The 
data on the others can be seen in Figure 10:2. 



-16- 

Figure 10 Teacher Dissatisfactions With the Televised Instruction for 
Pupils — the Entire Curriculum « (Data from first semester, 
1965.) 
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The check-list also made it possible to tell just what the teachers' 
complaints were about each course. Examples for two courses are shown 
in Figure ip:3, in the form of bar graphs. These tell a story about each 
course. The overall height of the bar graph "skyline" gives a rough idea 
of how frequently the teachers complained about the course on all counts, 
and the shape of the "skyline", or the differences in the heights of the 
bars, indicate which complaints were most frequent for each course. With 
13 courses, there would be 15 bar graphs. This kind of data can be looked 
at separately for each course by those particularly concerned idLth it, such 
as the television teacher and director, or the data for several courses can 
be compared — to find differences among the courses. In turn, these 
differences can be used as guideposts for eliminating complaints, by indi- 
cating which courses have been relatively successful in avoiding complaints 
of any one sort and opening the way toward a consideration of the devices 
they used. 

To obtain a more concise picture of how the courses fared in compari- 
son with each other on the various complaints, the data for the courses 
individually can also be looked at in conjunction with the data on the 
overall complaints for the entire ETV curriculum. The data on the over- 
all complaints can be used as a standard, and for each kind of complaint 
the per cent for each course can be sho^im as a deviation from, that of 
the average, An example of this appears in Table for three courses 

and four kinds of complaints, A plus sign (+) means that a course received 
the number of percentage points shown in excess of the average, and a minus 
sign (-) means that a course received the number of percentage points shown 
below the average. This makes it possible to see at a glance which courses 
received a particularly high or low number of complaints of any one kind. 
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*Since this is an example of the way data might be presented, the two 
courses are not identified. 



